A Learning Rate Analysis of Reinforcement Learning Algorithms in Finite-Horizon

Many reinforcement learning algorithms, like Q-Learning or R-Learning, correspond to adaptative methods for solving Markovian decision problems in innnite-horizon when no model is available. In this article we consider the particular framework of non-stationary nite-horizon Markov Decision Processes. After establishing a relationship between the nite-horizon total reward criterion and the average-reward criterion in nite-horizon, we deene Q H-Learning and R H-Learning for nite-horizon MDPs. Then we introduce the Ordinary Diierential Equation (ODE) method to conduct a learning rate analysis of Q H-Learning and R H-Learning. R H-Learning appears to be a version of Q H-Learning with matrix-valued step-sizes, the corresponding gain matrix being very close to the optimal matrix which results from the ODE analysis. Experimental results connrm that performance hierarchy.